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(57) Abstract 

A display screen (201) provides an image of a set of surfaces, e.g. a polyhedron (204), each one of the surfaces depictiiig a menu 
option. A remote control device is provided for providing an input signal to the television (201), which responds to the input signal by 
manipulating the orientation of the surfaces, and exposing the various menu options available to the user. The user can then click on the 
desired face of the polyhedron, corresponding to the desired option. In one embodiment, the remote control device comprises sensing means 
for detecting the motion and/or position of the remote control device. The polyhedron moves in a manner that tracks motion of the remote 
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10 Cross-Reference to Related Applications 



11 



This patent claims priority based on U.S. Patent Application 09/344,442, filed 



12 June 25, 1999; 09/378,184, filed August 20, 1999; 09/378,270, filed August 20, 1999; 

13 and 60/1 1 8,505, filed February 3, 2000, each incorporated herein by reference in its 

14 entirety. 

15 Background of the Invention 



1 8 television. One type of remote control device comprises a numeric keypad for punching 

19 in the number of a desired television channel, as well as buttons for selecting other 

20 options. Other types of remote control devices move a cursor on a screen to select a 

2 1 desired option. These techniques do not usually permit a viewer to preview a channel 

22 option before that option is selected. 

23 Another type of opfion selection scheme is to provide a set of small pictures on a 

24 television screen, and permit a user to "click" on one of the pictures to select an option 

25 corresponding to that picture. Such small pictures are sometimes called "thumbnails." 

26 Unfortunately, one can only put so many thumbnail pictures on a screen due to limited 

27 resolution of the television screen. 



16 



This invention pertains to remote control devices for controlling a television. 



17 



There are numerous types of remote control devices used for controlling a 
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1 It is an object ofour invention to provide an improved method and apparatus for 

2 selecting options for controlling an image display device, e.g. a television, computer 
^ screen, video editing device, or other type of device comprising an image display 

A 

^ Su[nmai:y 

(, A method \n accordance with one aspect ofour invention comprises the step of 

7 displaying midtiple video streams on a display device such as a computer monitor or a 

K ielevision. In one embodimeni, the screen of the display device contains a primary 

9 porlion and a secondary portion. A first one of the multiple video streams (hereafter (he 

10 ^*main video stream") is displayed on the primary portion of the screen. The second 

1 J porlion of the stream displays an image containing a plurality of geometric surfaces. In 

12 o^ie embodiment, the plurality of geometric surfaces are arranged as a polyhedi on At 

i.i least one of the multiple video streams is mapped onto at least one of the faces of the 

(4 polyhedron Typically, several of the multiple video streams are mapped onto associated 

1 5 ones of the faces of the polyhedron Alternatively, other faces of the polyhedron display 

16 images such as icons corresponding to an option that an operator can exercise, e. g 

1 7 turning the volume of a television up or down, changing a channel, or performing various 

18 video ediling functions. 

IV In accordance with another aspect ofour invention, a hand-held remote control 

2i) device permits a user to manipulate and/or select the video images mapped onto the 

2 1 geometric surfaces. In one embodiment, the position in which the rernot:e control device 

22 IS held is associated with the position of the geometric sm faces in the world coordinate 
2."* system. (As explained below, the term "world coordinate system" pertains to the 
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1 orientation of an image displayed on a display screen.) Thus, by rotating t lic ronote 

1 control device, one can rotate those geometric surfaces, 

S 111 one embodiment, the main video stream is mapped onto a flat geometric 

4 su) face. By rotating the remote control device, one can also rotate the Hat geomelric 

s surface upon which the main video stream is mapped. 

0 In accordance with another embodiment of the invention, the hand-held remote 

7 control device has a scrolling wheel. By rotating the wheel, one can rotate one or more 

8 of the geomelric surfaces upon which images are mapped. In one embodiment in which 
V liic plurality of geometric surfaces form a polyhedron, by rotating the wheel, one can 

10 rotate Ihe polyhedron. By rotating the wheel, one can also rotate the surface upcni which 

I ! the main video stream is mapped 

12 In accordance with another embodiment of the invention, the had-held remote 

\'s control device comprises a track ball. By rotating the track bail, one can rotate one or 

14 more of the geometric surfaces upon which images ai e mapped. In one embodiinejU on 
which ttie plurality of geometric surfaces form a polyhedron, by moving the track ball, 

16 one can rotate the polyhedron. By moving the track ball, one can also rotate the surface 

1 7 upon which the main video stream is mapped 

18 As mentioned above, in one embodiment, video streams are mapped onto the 

IV various faces of a polyliedron, and rotating the hand-held remote control device results in 

:>() rotjitiori of the polyhedron (In one embodiment, the polyhedron can rotate about only 

21 one axis. In another embodiment, it can rotate about more than one axis.) A control 

22 element on the remote control device, e.g. a button or switch, can be used to select which 
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I image on a polyhedron Face is to be shown on the primary portion of the disphiy device 

1 as tlie main video stream. 

3 \\\ one enibodimenl, the hand-lield remote control device is held by a user who 

4 can rotate I he remote control device, e.g. about any desired axis. Means ai e provided 

.s within ihe remote control device for sensing motion aJid/or position of the remote control 

6 device, and communicating to a receiver within the television lhat motion and/or position 

7 A receiving circuit within tlie television causes the image of the polyhedron to rotate or 
H move in a manner that mirrors the motion of Ihe remote controller. Wher» a face of the 

polyhedron depicting an image representing a desired option is facing the user, he can 

i(j actuate a button or other control device on the remote controller to select that option 
1 1 I n anolhei- embodiment, instead of displaying a polyhedi on, the menu options 

\l displayed on the video screen can be displayed in another form. However, different 

! '] menu options can be displayed and/or selected in response to the motion and/or position 

1 't of the remote control device. 
J ^ 

I likit^f !-^MCQj>li.ori. o 

1 7 figs. 1 A to IE illustrate the operation of a 3D graphics pipeline, 

t X Figs 2A and 2B illustrate manipulation of a 21) image. 

fig. 3 is a simplified block diagram of a personal computer (PC) coupled to a 

20 graphics controller with a 3D graphics pipeline. 

2 1 Fig 4 illustrates a television displaying an image of a polyhedron constructed ni 
T). accordance with our invention. 
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I Fig. 5 illustrates a remote control device for controlling the television comprising 
'I position or motion sensors. 

^ Fig. 6 illustrates a remote control device for controlling the television comprising 

i a track ball. 

.s Fig. 7 illustrates a remote control device for controlling the television comprising 

u a rotating wheel. 

7 l ig. 8 illustrates a television displaying a band of images. 

X 

.QetailedJ)^^^ 

10 As mentioned above, a method in accordance with our invention involves 

I I displaying an image of a polyhedron on a television or other display device. Each face of 
12 the polyhedron depicts an image representing an option that can be taken by someone 

1 i operating the television. 

\ -\ The polyhedj on and the images on the iaces of the polyhedron are generated 

l,s using a 3D graphics pipeline in a novel manner. In order to explain the manner in which 

16 the polyhedron and images are generated, we will first explain how a 3D graphics 

17 pipeline is normally used. We will then describe its use during a method in accordance 

18 with the invention. We will then describe remote control devices that can be used to 
(0 numipulalc the orientation of the polyhedron. 

20 

2 J 3D G rap hics Pipelines 

^? The 3D graphics pipeline referred to \w this patent can be implemented by a 

23 combination of hardware elements, known as accelerators, and software, some of which 

24 is sometimes referred to as drivers. The partitioning between hardware and sofHvare may 
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1 vary, depending upon the CIHJ used and the graphics card in the system, but the overall 

I sysl eni performs the method steps described below. Portions of the pipehne tasks can be 

t performed by soUware, which is less expensive than hardware, but in general slower than 

1 hnrdware solutions al the present lime The hardware and software that perlbrm the steps 

5 described below are referred to simply as a pipeHne, without regard to the specific 

0 partitioning. 

7 flic following is a simplified, general description of 3D graphics pipelines. It is 

K not intended to describe any specific product (e.g. products mentioned later in this 

9 patent). Rather, the following description is merely a general explanation of 3D graphics 

H) pipelines to assist the reader's understanding. 

\ I Currently, graphics objects created using a 3D graphics pipeline can be described 

1 2 as a set of geometric surfaces. One way of constructing a geometric surface in a graphics 

1"^ pipeline is to create a'^mesir of "primitives." A ''primitive" is a small geometric surface 

M thai can be defined by a set of vertices. For example, the primitive can be a polygon (e.g. 

\^ a triangle or quadrilateral) defined within the pipeline in terms of the locations (in y 

K) and z coordinate space) of its corners or vertices. A set of several primitives is used to 

1 7 i\Gf\m a larger 3D surface. 

>H Instead of using primitives, such as polygons, some graphics pipelines can 

ly process geometric surface areas defined in other ways, e.g. by mathematical equat ions 

20 This technique for defining geometric surface areas is called "implicit.'' As explai ned 

2 1 below, both techjiiques for defining such surface areas can be used. 
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I For purposes of clarity orcxplaiiation, we will first describe a grapliicy pipeline 

1 lhat processes geometric surface areas using triangular primitives. Other types of 

3 graphics pipelines will be discussed later on. 

4 In this first example, a 3D graphics pipeline constructs a 3D image of^an object 

f) Irotn a 2D pixel array (lypically called a ''texture map"). Fig. 1 A illustrates a 2D image 2 

6 of a set of "textures." (As will be explained below, this texture map is used to create the 

7 image of an object in this case, a house. Image 2 includes a portion 2a, wliich has the 

8 appearance of bricks, portion 2b, which has the appearance of roof shingles, poition 2c, 
V which has the appearance of a door, and portion 2d which has ttic appearance of a 

10 window.) 2f) image 2 is stored in a digital memory in the form ofan array of pixels. 

I I Hach localion in the memory stores a pixel, which is one or more words of data indicating 

12 the color, color saturation and briglitness corresponding to that pixel T he location of (he 
1 pixels wilhin the array is typically referred to as u, v coordinates (not to be confused with 

14 Lhe Y, IJ and V signal names used to describe certain video signals), ('I^he u, v 

15 ctH)rdinafes are similar to y coordinates of the Cartesian coordinate system. In Fig. 1 A, 
\<^ the pixel array is an n by m array, where n and m are integers.) 

''^ mentioned above. Fig. I A represents a pixel array. Physically, the array 

l^i comprises data loaded into a memory. 

The next step in the process is to provide or prepare a geometric surface. In this 

20 example, (he geometric surface is in the form of a mesh 4 of primitives 5 in three 

21 dimensional space (Fig. I B), in the case of Fig IB, the primitives are triangles, but other 
n types of ()c)lygons can be used. The mesh of primitives represents a three-dimensional 

^3 sJiape ofan object 0 in 3D space (in the case of Fig. IB, the shape of a house). The 
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1 position of each vertex of each triangle within mesh 4 is stored in a memory in the ibrm 

2 oi X, y and Cartesian coordinates, relative to the object. These coordinates are 
.sonielimes referred to as model coordinates ('*MC"). The process of preparing such a 

4 inesh is well -known, and described in standard graphics libraries, such a?; Real 3D, 

-i published by Real 3D, a Lockheed Martin Corporation, in 1996, and Direct 3D, published 

6 by New Riders I*iiblishing in 1997 

7 1 he mesh of Tig. I B is not displayed as such. Rathei', the mesh o f Fig 1 13 is a 

8 rcpresenlation of what is stored in a digital memory. Specifically, the memory siores tlie 

9 locations, in terms of y and z coordinates, of each vertex within niesh 4 

10 The next step is to map or ''bind" the two-dimensional texture map of l-ig. I A 

1 1 onto mesh 4 of Pig. I B. This is accomplished by mapping each triangle vertex to a 

1 1. location in tlie texture map. In effect, a list of data points is prepared that associates each 

[3 vertex of mesh 4 to the u, v coordinates of a particular point (pixel) in the texture map of 

14 Fig. 1 A. (The locations in the texture map to which the vertices are bound are sometimes 

1 5 referred to as "control points.'') 

16 This portion of the process is roughly analogous to an upholsterer choosing a 

17 piece of fabric, and binding it with a few nails to the corner of a couch beiJig upholstered 
\Vi {lite nails are like control points). The upholsterer subsequently asks his apprentice to 

19 finish attaching the fabric to the couch. In this case, the 3D graphics pipeline finishes the 

2(1 t nsk instead of an apprentice. 

1 T'igs t A and IB describe the process by which one texture map (Fig- 1 A) is 

22 mapped onto one mesh 4 representing one object 0. A graphics pipeline can, and often 

2.1 does, map one or several texture maps onto the same or several different objects 
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10 



Hie next f5tep in the process is lo set up a world coordinate model'' of the varioiis 

2 objects Lo be displayed. This requires establishing a position and diiectional orientation 
for each object lo be displayed. Vor example, supposing that there are to be two objects 

4 to be viewed; a tetrahedron T and a cube C (Fig. IC). During this siep of the process the 

^ pipeline is instructed that cube C is to be lacing in a certain direction, and is to be located 

6 partially in (ront of tetrahedron T relative to a certain frame of reference. Again, the 

7 structure of Fig \C is not displayed per se. Rather, the graphics pipeline sets up 
« processing of the model coordinates in accordance with the parameters of the position 
9 and oi ientation of the object. 

The next step is to select a frame of reference, (''or example, it might be decided 

1 1 that the ' viewer'' will want to observe the objects from a position corresponding to a 

n corner of the world coordinate model (e g, position P in Fig. I D). Thus, a virtual 

I ^ vicAvpoim, viewing direction and aperture will be selected The parameters associated 

M with this "viewer" define the screen coordinate (SC) system. Further, it might be decided 

I the viewer will observe these objects with a light source located at a position I., 'f he 

16 graphics pipeline will set up another processing pipe to process the world coordinate data 

17 into the soeen coordinate data which will cause a computer screen lo display the image 
i« as it would be perceived by the observer at position P (e.g. the image of Fig. I D). I n 

IV other words, the computer screen will provide an image of tetrahedron T and cube C! as 

20 they would be observed by a viewer if he were standing at position P, and a light source 

2 1 were present at location L. This image will be provided initially as a pixel array in a 

22 frame buffer and then displayed by the computer screen. The image in tl»e frame buffer 

23 is refreshed, i.e. regenerated according to the specifications programmed into the 
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/ pipeline, typically al about 50 to 120 tinies per second. There are jnany differcru 

2 niethotls Ibi optimizing the pipehne, and minimizing the time spent processing ihc 

:\ invisible parts of the objects, such as the backside of cube C facing away from the 

4 viewer Such details are well-known to those <;kil]ed in the art, and will not be discussed 

r> in detail here. 

During the above-described process constrticting the pixel array and providing il 

7 in the frame bulfcr, the pipeline a) fetches the portion of texture map 2 "tacked'' to the 

H vertices of mesh 4 (and therefore stretched over each triangle); b) determines how and 

9 where tlial portion of the texture map should appear, given the orientation of the triangles 

10 relative lo the viewer and the location of the light source, and c) constructs the 

i I appropriate bit map pixel array for storage in the frame buffer. The contents of tliis frame 

1 2 bulTcr are then displayed as an image on a computer screen. 

t.'i Thereafter, the 3D graphics accelerator permits one to manipulate the displayed 

14 objects in any desiied manner. For example, if one wants to rotate the image of 

15 telrahedron T by 45^' (Fig \ the 3D graphics accelerator facilitates this manipulation 

16 fhis is accomplished by providing a new set of parameters in the world coordinate model 

17 lor the graphics pipeline indicating the new position and orientation for tetrahedron \ . 
IH Aller this occurs, the next time the graphics pipeline regenerates the image stored in the 
\') Iranic buffer, the regenerated image will reflect this rotation of tetrahedron T, 

Similarly, suppose that it is desired to display whal would appear to the viewer if 

21 he took ten steps forward from his location at position P. Tlie next time the grapfiics 

22 pipeline regenerates the image, it will generate and store another pixel array in the frame 
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I l)uiTer corresponding to what would appear to such a viewer, and this pixel arr^^y is 
provided as another image on the computer screen. 

I h is Thus seen that the graphics pipeline is extremely useful in apphcations such a 
.1 video games, where it is desired to simulate what would appear to a game player if he 

^ were wandering past a set of objects. 

As mentioned above, some graphics pipelines create models ofgeoitietric surlaa 

7 using an implicit technique. These surfaces are often described as a function of tite 

X position coordinates, i.e. f (x.y.z), or can also contain some vertices. Control points and 

<j additioncil formulas associated with such surfa<:es are used to bind a digital pixel array 

10 (e.g. an array as shown in Fig. t A) to the implicitly defmed surface, and the process 

I I proceeds as described above. The major difference is thai instead of defining surface 
\2 areas in terms of primitives with vertices, the surface areas are defmed in terms of 

\} mathematical equations. 
14 

!5 MHJlipuJaljpnpXl 

\(, As mentioned above, one embodiment of the invention is a remote controller 

17 which cooperates with video graphics circuitry that provides an image of a polyhedron, 

iH each face of the polyhedron displaying an image corresponding to a menu option. I wil 

19 now explain how that image is provided. In particular, 1 will describe a method for 

20 manipulating a two-dimensional image 

2 1 A method for manipulating a two-dimensional image begins with tlic step of 

22 obtaining a two-dimensional digital image (e.g. image 10 in Fig. 2A). This step can be 
2.^ perlbrmed, e.g., by scanning an image such as a photograph or other picture using a 
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1 conventional digital scanner. Tlie digital image can also be obtained from a convcnlional 

2 digital camera The image can also consist ofdigital video image, e.g. out ofa live oi 
^ stored video streani, which is basically a fast succession of 2D images However, any 

4 other source of n 2D digital image can be used. I^he digital image is typically stored in a 

(Tiemory as an array of digital values In one embodiment, the digital values are in ii 
f) compressed form, e.g. using a compression technique such as MPEGI or IVIPKC}2 or 
7 other f(ji mats. In the case of compressed digital values, they must first be decompressed 
X prior to processing. Also, scanned images or digitized images from any soujce such as 

9 cable TV, an antenna, cameras, etc. can be us<;d. 

10 As mentioned above, for the case of video images, dozens of frames per second 

i 1 comprising millions of pixels per second musi be processed. We discovered that slandard 

12 graphics pipelines can be used to process frames of data sufficiently fast to process video 

i:i images 

^n.y typt^ of memory can be used to store the digital 2D image, e.g. 

1 5 semiconductor memories (SRAMs, DRAMs or other semiconductor memories), a 

10 magnetic memory (e.g. a hard disk, a floppy disk, magnetic tape, or magneto-oplic disk), 

1 7 or other type of memory device {e.g. an optical disk). The pixels corresponding to the 

l« stored image can be stored in terms of RGB values (e.g. the strength of the red, green and 

1** blue components of the pixel color), YIJV values or other values. (For YIJV values, Y 

^-0 corresponds to the amplitude or brightness of the pixel value, U corresponds to the color 

2 1 and V corresponds to ihe saturation ) The pixel values can be encoded in olho" ways as 

2?. well. Depending on the situation, a conversion may be required before further 

■2.^ processing. 
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1 Next, ii 3D graphics pipeline is set up. This is accomplished by providing 

2 inslmclions to the 3D graphics pipeline as to what is to be done with the data thai is to be 
'\ provided. Setting up graphics pipelines per se is well known in the art, e.g. as described 

4 in the Microsoft Direct 3D SDK (soflware developer kit) or Direct 3D 

5 Thereafter, a computer model of a planar geometric surface is generated. 'T'his 
(> computer model can comprise a set of primitives, e.g. polygons such as triangles Id 

7 anotlier embodiment, the computer model can comprise an implicit description of a flat 

K geometric surface. This implicit description is typically a mathematical function (e.g. a 

') function of y and i) as described above. 

10 Fo)- the case in which the planar geometric surface comprises a mesh of 

1 1 primitives, the number and shape of primitives and the type of primitives can vary Fig 
i;. 2B illustrates a mesh 12 that can be used to practice a method in accordance with our 

i:. invention. Mesh 12 is similar to mesh 4 described above. However, unlike mesh 4, all of 

\A Ihe vertices of inesh 12 are coplanar (or substantially coplanar). In one embodimeni:, 

l.S inesl» 12 comprises about 5000 triangles, which would be acceptable tbr processing a 

16 video image. Of course, other numbers of primitives could be used. 

17 After constructing the planar geometric surface (e.g. iYiesh 12)., image 10 is 

18 mapped, or bound, onto the flat geometric surface. This is accomplished in the following 

19 way. For ihe case in which the flat geometric surface is a mesh such as mesh 12, each 

20 vertex of the ilat geometric surface (e.g. the triangle vertices) is associated with an image 

21 pixel location (i c. control point). Thus, each control point is associated with a texline 

22 coordinates (u, v) corresponding to a pixel. A table of data listing each vertex and its 

2.^ associated u, v texture space coordinates is set up. This is called "binding/' (See Kamen, 
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1 WiEE Computer Socicly, IEEE Computer Graphics and Applications, Jan -^Feb, 1997, 

2 Vol 1 7, No I .) For the case in which an implicit technique is used to define the Hal 

< geometric surface, control points within the implicitly defined surface are bound lo pixel 
4 array coordinate space (u, v coordinates) in a manner analogous to the triangles discussed 
^ above 

6 After image 10 is mapped into mesh \2, the object can be manipulated by 

7 manipulating the world coordinates The world coordinates describe where in the x, y, /. 
X space the textured plane is to appear, and whai its orientation will be (i.e. what angle it 

9 should be held ai with respect to the viewer). In addition, the screen coordinates foi the 
10 objecl can be changed. As a result, when the 2D textuied iiTiage is finally piepared, it can 
J I be prepared in such a manner that reflects the desired manipulation. For example, il can 
1 2 be rotated about any axis, magnified, shrunk, etc. 

After establishing the world coordinate model and screen coorditiate model, the 
pipeline prepares an array of pixels in the output Irame buffer (OFB), including pixels 
1.^ showing the manipulated textuj*ed mesh 12. The array of pixels in tJieOFB is displayed 
16 on a CRT or other type of screen. 

One can manipulate the video image by, for example, changing the world 
H coordinate parameters, e.g. telling the pipeline to tilt the video image about any axis 
*j (including an axis perpendicular to the screen or in the plane of the screen). Thus, when 
ilie pipeline regenerates the pixel array in the OFB, the regenerated video image will 
appear lilted about the selected axis. Since the pipeline will regenerate the image a( a 
preprogrannned rate according to the system used, live video will appear as live video. 
Thar IS because every time a new pixel array is generated, t he texture map, which 
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contains the incoming video frame buffer, is reread and put thiough the pipeline. .Siace 
the texture mapping process also contains features for pixel interpolation, an automatic 
icsoltition adaptation occurs 

One can bend or warp the image by moving ihe vertices about which the image is 
mapped Thus, one can alter llie Hal geometric plane of Fig. 2B to thereby warp the 
image. When the pipeline regenerates the pixel array in tiie frame buffer, the image will 
appear warped. 

One can move the vertices so that mesh 10 becomes a cylinder When the 
pipeline regenerates the pixel ariay in the frame buffer, the image will appear wrapped 
around a cylinder (Of course, the mesh 10 can be altered into other shapes, and the 
image would be wrapped around the other shape.) These modifications could be done al 
a speed that would create the impression in the viewer that the image was being wrapped 
01 warped gi adually 

One could magnify or shrink images by moving vertices away from or closer to 
each other, or moving the image closer or further from the viewer in the world coordinate 
sy.stem. or by re-parametcrizingtlie model coorriinatcto world coordinate conveision. 

Ha..a.lmr.ejmdSof^^^^ 

One embodiment of our invention can be practiced using a PC having the 
following. 

I A CPU such as a Celeron or Pentium, e g. as manufactured by Intel, or a K6 

processor, e.g. as manufactuied by Advanced Micro Devices. 
2. 32 MB of memory or greater. 
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I 1 A :U) IIW adapter. This is a iype of graphics card currently available on t:he 
- niarkel. The JD MW adapter should have 4 MB oJ^ memory (preferabJy 8 MB) 

.'i and an advanced graphics port (AGP) interface. (An AGP interface is a type of 

4 bus standard that is well-known in the art.) Alternatively, a peripheral connection 

5 interlace ("PCT") can be used in lieu of a AGP. 1'he PCI is a type of bus standard 
^^^'^^^ '-^ well known in the art Examples of appropriate 3D IIW adapters include 

7 the rNT^2 available from Riva, the A TI Rage 128, the Matrox G4()0, the lYidenl 

« Blade 3D and the S3 Savage. 

4. ^I'he operating system can be Windows 95, Windows 98, Win2000, or any other 
It) operating system that supports direct 3D. The Windows operating system 

1 1 includes a standardized platform called Direct X for Windows. 

''^ I'^ "ne embodiment, a usei sets up the flat geometric surface (lor example, a 

i :^ triangle mesh) in the Direct 3D windows environment. The set of instructions is then 
14 provided to the graphics pipeline, which finishes the rendering process. Howevei , in 
1-^ another einbodimenl, the PC comprises a bypass mechanism that permits one to access 

16 ihc hardware accelerator directly using a software interface provided by the graphics cai d 

17 manufacturet. 

Tig. 3 is a block diagram of a computer system 50 for generating images thai can 
be used in accordance with the hwention. Referring to Fig. 3, system 50 comprises a 
2i) CPU 52, e.g. a Pentium II class CPU, comprising a cache memory 52a, a core 52b and an 
2i internal bus 52c for facilitating communication between core 52b and cache 52a Core 
:^:> 52b communicates via a CPU bus 54 to a system controller 56. System controller 56 
2 i communicates with the system memory 58 via a memory bus 60 System memory 58 
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1 iitcludes a first portion S8a which stores system memory programs and a second portion 

2 58h iliat stores tlie texture jnaps such as described above. 

:\ Also included in system 50 is a PCI bus 62 for fiicilitating communication 

4 between system controller 56 and I/O devices ()4, 66 and optionally a disk drive 68, I/O 

s device M can be any type of I/O device, In one embodiment, I/O device 64 is a network 

r> interlace adapter "NIA'' for receiving signals from any type of network, including but nol 

7 limited to satellite broadcast, cable broadcast, telephony, fiber and topologies such iis 

K Wide Area Networks (including the Internet), Local Area Networks, Local Multiple Drop 

y Networks etc. some embodiments, I/O device 64 can be a inodem, and in others 1/0 

10 device 64 is a tuner for receiving television signals etc. 

11 In one embodiment, 170 device 66 is a video capture card with a driver. Data 

12 froni the video capture card is either loaded by DM A (direct memory access) or CPU 52 
1.3 into a IVame buffer, typically within main memory 58. However, the frame buffer may be 

14 in other memories within system 50. In some embodiments, multiple video streams or 

15 image sources are available, such a.s local storage, capture card 66, Nl A 64 or other, not 
IT) explicitly shown sources 

17 Syslcm 50 also includes an AGP graphics controller 70 comprising a 31) 

IK accelerator. In one embodiment, AGl^ graphics controller 70 communicates with systen) 

10 controller 56 via an AGP bus 72. In an alternative embodiment, AGP graphics controller 

20 70 can communicate with system controller 56 via PCI bus 62 (e.g. as shown in pliantom 

21 in Fig 

22 Graphics controller 70 uses its own local memory 74 to generate and store pixel 
\ arrays to be displayed on a video display unit 76. 
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1 It is emphasized thai system 50 is only one example of a system that pevforrns a 

2 method in accordance with our invention. Other hardware can be used as well. 

.1 The above-mentioned method can be used to manipulate image streams sucli as 

4 television images. This method is particularly appropriate since video images coniprise a 

5 succession of frames at a rate of about 60 frames/second in North America. F'or instance 
(^ in the case of NTSC, about 9. 1 Mbytes per second throughput are required. (N'f'S(.' is an 

7 abbreviation of ''North American Television Standard for Color. It is llie standaid used 

8 for television signals in North America.) 

') The system of Fig. 3 can move and tilt portions of different video images oi other 

10 images onto different portions of a screen such as <\ television screen hi one 

1 ! embodiment, tJie images are transformed to appear on the faces of a polyhedron (e.g. ii 

12 cube). As explained below, the polyhedron is used as a new type of television menu 

n option display. In particular, a novel remote control device (described below) permits a 

\4 user 10 turn the polyhedron to see the different images on the various faces of the 

15 polyhedron. Ailer the polyhedron is turned in an appropriate orientation, one can ''click" 

16 on a desired polyhedron face, or a portion of a desired polyhedron face, to select a desired 

17 option. 

iK The manipulated image provided in accordance with our invention can be 

provided !o any appropriate output device, e.g. a television scj een, a video projector, a 

20 [ lf)TV monitor, or a PC screen, 'f he image manipulated in accordance whh our 

21 invention could come from any of a number of sources, e.g. an analog or digital video 

22 input, a cable TV input, a satellite input, the internet, a digital scanner, a digital camera, 
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I or numerous other sources. (In the case of an analog input, one would first digitize the 
'I image.) 

jvjg 4 illustrates a television 200 in which the television screen 20 1 is divided into 

7 a primary portion 202 and a secondary portion 203 Primary portion 202 displays a 

H i^rimary video image. Secondary portion 203 depicts a polyhedron 204 in accordance 

9 with the invention, t elevision 200 includes a controller 205 for generating an image of 

10 polyhedron 204, 'fhis controller can include Ihc hardware elements depicted in Fig 3. 

I I the faces of polyhedron 204 can depict video images, e.g. images of what appears on the 
12 various television channels, A given face of polyhedron 204 can also include both video 

images and additional information (e.g. in the form of alplianumeric characters or icons), 
eg. ihe program name, channel number, etc, In addition, the faces of the polyliedron can 

15 depict icons concerning various options, e.g. options related to television volume, on/oti 

16 switches, control of a VCR, options related to editing video images, etc. (Images 

17 corresponding to such icons are stored in a memory within television controller 205.) 

1 8 Any appropriate screen and display technology can be used for television 200. 
l*J 7'elcvision 200 is controlled by remote control device 206, which communicates with 

20 television 200 by emitting a signal, e.g. an infrared, radio or other type of signal that can 
'I I be transmitted and received. (Remote control devices that communicate with a television 
22 using infrared signals are well known in the art. See U.S. Patent 4,9 18,439, for example.) 
2"> In lieu of, or in addition to emitting an IR or radio signal, remote control device 206 caJ^ 
24 be connected to and communicate with television 200 by a wire. 
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, . m one embodimeat, lemole control device 206 has the capability of sensing 

2 mot ion, e.g. as symbolized by arrow 207, indicating rot^uion of device 206. Such rotation 

i is sensed, e.g. by techniques described below. Signals indicating such rotation arc 

4 cornn^unicated to a receiver within television 200, which in turn sends commands to 

5 controller 205. A 3D pipeline within controller 205 orients polyhedron 204 in a manner 

6 that mirrors motion of remote control device 206, either identically or partially Fo) 

7 example, the signal depicting motion of remot<j control device 206 can be filtered lo 

8 eiimiiiate jerking movements. 

9 Fig. S illustraifis several features of remote control device 206. Element 21 1 is a 

10 transmitter for communicating with television 200, Element 21 1 is typically an l.l< 

1 1 transmitter, but it could also be an ultrasonic, radio, magnetic induction, or other type of 
J 2 non directional communication device. Since IR is somewhat directional, more than one 
1 1 emitter may be used in order to guarantee proper communication while handling remote 

14 control device 206. 

15 Also shown in Pig. 5 are a set of buttons and displays, e.g. light emitting diodes ot 

16 liquid crystal displays, possibly trackballs, etc., symbolized as three fields 212, 213 and 

17 21 4. Also shown in phantom are a battery 220, a printed circuit board 22 1 (containing a 
IX microcontroller 222 with built in program store), and two motion detectors 230a, 230b 
1.^) By calculating the difference in motion of these two detectors 230a, 230b, 

20 microcontroller 222 can determine the relative motion of detectors 230a, 230b as well as 

1 1 the direction of motion The two motion detectors 230a, 230b thus permit 

22 microcontroller 222 to determine which way device 206 is turned. In response to such 

23 motion, microcontroller communicates to controller 205 the manner in which remote 
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1 control device 206 has been manipulated. ControlJer 205 alters the image of polyhedron 

2 204 appropriately. The user of remote controller 206 selects an option corresponding to 
an image facing the user by pressing au appropriate button the remote control device. 

i (Microcontroller 222 reads or senses the various buttons and other input devices on 

s remote control device 206, and provides appropriate signals to controller 205 in response 

() thereto.) 

7 As mentioned above, controller 205 w ithin television 200 causes motion of 

S polyfiedron 204 to mirror motion of controller 200. Each face of polyhedron 204 depicts 

'J one or more menu options that a user can select by pressing appropriate buttons on 

10 remote control device 206. In this way a user can select a television channel, increase or 

i I lower volume, turn the television on or off, select a signal source for the television (e.g. 

1 2 selecting t)etween cable TV, a VCR or the internet), etc. In addition, one can cause an 

i .1 i mage on one of the polyhedron fiices to appear on main portion 20t of the television 

14 screen 

I ^ Iti lieu of motion detectors 230a, 230b, other position or motion detection devices 

16 can be used, e.g. gyroscopes, GPS (global positioning system), or other inertia or position 

17 tracking devices. 

IH h is noted that remote control device 206 is different from other types of remote 

10 control devices. Tor example, while trackballs cooperate with mechanical structures for 

20 sensing the motion of the trackball, the trackball can only be used with the ball 

21 mechanically resting against those structures. Remote control device 206 can sense its 

22 own motion although it is not mechanically tethered to other sensors, or does not 

23 mechanically rest against other sensors. In particular, remote control device 206 does not 
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1 rcqiiiie a stationary tion-moving component lo determine the motion of remote cont rol 

2 device 206. 

:\ Remole control device 206 is a preferred structure for manipulating polyhedron 

4 204. However, other structures can be used for manipulating polyhedron 204, e.g. 

control buttons or track balls Fig 6 illustrates a remote control device 240 comprising a 

u track ball 242. During use, an operator rotates track ball 242. The rotation of track ball 

7 242 is sensed by remote control device 240, and a signal is communicated by device 240 

K to controllei- 205 within television 200, which causes polyhedron 204 to rotate. 
V \\\ another embodiment, a remote control device 244 is used which comprises a 

10 rotating wheel 246 (Fig. 7). During use, an operator rotates wheel 246, which is sensed 

1 1 by remote control device 244 A signal is thus communicated by device 240 to controller 
I?. 20."^ within television 200, which causes polyhedron 204 to rotate. 

n \\\ the above-mentioned embodiments, video streams are bound to the various 

14 geometric surfaces forming the polyhedron. The polyhedron is rotated by ahering the 

15 world coordinate system that is applied to the 3D pipeline. Jn one embodiment, CI^IJ 52 

16 within controller 205 determines what change is to be made to the world coordinate 

17 system to in response to the signaJ controller 2t)5 receives from remote control device 

18 206 {or j etnote control device 240 or 244 as the case may be), 

1*^ Jn one embodiment, the image on the primary portion 202 of television screen 20 1 

20 is also bound lo a geometric surface by a 3D graphics pipeline. One can rotate or 

21 manipulate the image on primary portion 202 of the television screen using the remote 

22 control device. 
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i Although Ihc gcoinelric surfaces in second portion 203 ofthe television screen 

1 (orm a polyhedron in tiie embodiment of Fig. 4. in other embodiments, the geometric 

'S surfaces do not form a polyhedron. 

4 In one embodiment, a band of images 204' is provided in second portion 203 of 

5 television screen 201 (Fig, 8) One moves band 204' by moving remote control device 

6 206 As band 204' moves (e.g. as symbolized by arrow 208), dilTerent images become 

7 visible. Vox example, image 204a on the far right of band 204' will disappear and image 

8 204b will take its place A new image will appear at the position of image 204c 

9 One can select an image (or a menu option represented by thai image) by 

l(» selecting an image that visually appears parallel to screen 201 of television 200 In other 

i I woids, by actuating an appropriate control button on control device 206, the image thai 

n appears parallel to screen 20 1 (typically center-most image 204d) is selected. In another 

\^ embodimeni, one can rr»ove a cursor (using a control button on control device 206) to 

1 \ point to a particular image within band 204\ and then actuate another control button to 

15 select thai image (or the menu option represented by that image). Alternatively, one oC 

10 the positions along band 204' can be highlighted or otherwise marked as representing an 
17 image to be selected One can move different images to the marked poshion to select that 
IS image. 

19 Band 204' can be either a "closed band" or an "open band " By "closed band" 1 

20 mean a band whereby scrolling band 204' far enough in one direction (e.g. rotating the 

21 band 360 degrees to the right) will eventually result in the same images being returned to 

22 their original position. This is to be contrasted with an open band of images in which 
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1 moving the images to the right, for example, will eventually expose a left^-mosl image, 

2 with no image exposed to ihc left of that left-most image. 

^ Selection of images on a face of polyhedron 204 can be accomplished in a manner 

4 similar to image selection for band 204\ 

5 While the invention has been described with respect to specific embodiments, 

6 those skilled in the art will appreciate that changes can be made in form and detail 

1 wilhoul departing from the spirit and scope of the invention. For example, instead of 

H using a polyhedroji, other mulli-face images can be used in the above-described manner, 

V Further, a plurality of polyhedra or bands can be depicted and manipulated on a television 

1(1 sci een Bach face of the polyhedra or bands can include two or more portions that can 

I I depict various options. As mentioned above, instead of using motion detectors within the 

1 2 remote control device, position detectors can be used. Different types of display devices 

I :s can be used in conjunction with our invention, e.g. CRT screens, LCD screens, or other 

1 4 display devices Accordingly, all such changes come within the invention. 
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1 I claim: 

2 1 . Method comprising: 

3 providing a display screen and a control device, said display screen displaying a 

4 plurality of surfaces, an image being depicted on each of said surfaces within said 

5 plurality of surfaces; 

6 actuating a control input to said control device; and 

7 manipulating the orientation of said surfaces in response to said control input 
8 

9 2. Method of claim 1 wherein said images on said surfaces are provided by a 

10 graphics pipeline. 
11 

12 3. . Method of claim 1 wherein at least some of said images comprise video streams. 
13 

14 4. Method of claim 1 wherein said images on said surfaces depict menu options, said 

1 5 method further comprising the step of selecting one of said depicted options. 
16 

17 5. Method of claim 1 wherein said control device is a remote control device that 

18 senses the motion and/or position of said remote control device. 
19 

20 6. Method of claim 1 wherein said control device is a remote control device 

.2 1 comprising a rotation wheel, such that a user can rotate said rotation wheel, said step of 

22 manipulating comprising the step of manipulating the orientation of said surfaces in 

23 response to said rotation. 
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2 7. Method of claim 1 wherein said control device is a remote control device that 

3 comprises a track ball, such that a user can rotate said track ball, said step of 

4 manipulating comprising the step of manipulating the orientation of said surfaces in 

5 response to said rotation of said track ball. 
6 

7 8. Method of claim 1 wherein said surfaces form a polyhedron, said act of actuating 

8 changing the orientation of said polyhedron. 
9 

10 9. Method of claim 8 further comprising the step of applying said images to said 

1 1 polyhedron with a grapiiics pipeline. 
12 

13 10. Method of claim 1 wherein said surfaces form a band of images. 
14 

15 11. Method comprising the steps of: 

16 displaying a set of images on a display screen, said images corresponding to a 

17 control option that can be exercised, such that the display of said images represents a 

18 display of a list of control options that can be exercised; 

19 actuating a control to thereby move said plurality of images on said screen, 

20 thereby changing the list of options being displayed that can be exercised. 
21 

22 1 2. Method of claim 1 1 wherein at least some of said images are video images. 
23 



wo 00/46680 PCTAJSOO/02870 

27 

1 13. Method of claim 1 1 further comprising the step of selecting one of said images, 

2 thereby selecting an option corresponding to said selected image. 

3 

4 14. Method of claim 1 1 wherein said images are arranged as a band of images. 

5 

6 15. Method of claim 1 1 wherein said images are arranged to form a polyhedron. 

7 

8 1 6. Method comprising the steps of: 

9 displaying a plurality of surfaces on a display screen, an image appearing on said 

10 surfaces; and 

1 1 rotating said images in response to actuation of a control device. 
12 

13 17. Method of claim 1 6 wherein the surfaces form a polyhedron. 
14 

15 18. Method of claim 1 7 wherein said screen comprises a main portion and a 

16 secondary portion, said polyhedron being displayed on said secondary portion, said 

17 method further comprising the step of selecting one of the images of said polyhedron and 

18 displaying said selected image on said main portion. 
19 

20 19. Method of claim 1 6 wherein the surfaces form a band of images. 

21 

22 20. Apparatus comprising: 

23 a display device displaying a plurality of images on a plurality of surfaces; and 
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I a control device, wherein actuation of said control device rotates said surfaces- 

2 

3 21. Apparatus of claim 20 further comprising a graphics pipeline for generating said 

4 plurality of images on said plurality of surfaces. 
5 

6 22. Apparatus of claim 20 wherein said surfaces form a polyhedron. 

7 

8 23. Apparatus of claim 20 wherein at least some of said images comprise video 

9 images. 
10 

1 1 24. Apparatus of claim 20 wherein each of said images depicts a menu option, and 

12 said control device is a remote control device containing a control for selecting one of 

13 said depicted options. 

14 

15 25. Apparatus of claim 20 wherein said control device is a remote control device that 

16 senses the motion and/or position of said remote control device. 
17 

18 26. Apparatus of claim 20 wherein said control device is a remote control device that 

19 comprises a rotation wheel, such that a user can rotate said rotation wheel, wherein said 

20 graphics pipeline manipulates the orientation of said surfaces in response to rolatioa of 

21 said wheel. 
22 
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1 27. Apparatus of claim 20 wherein said control device is a remote control device that 

2 comprises a track ball, such that a user can rotate said track ball, said step of 

3 manipulating comprising the step of manipulating the orientation of said surfaces in 

4 response to said rotation of said track ball. 
5 

6 28. Apparatus comprising: 

7 a display screen for displaying a plurality of images, each of said images 

8 corresponding to a control option; 

9 a control device for moving said plurality of images, whereby different sets of 

10 images corresponding to different control options can be displayed on said display 

1 1 screen, said control device also comprising a control element for selecting one of said 

12 options. 
13 

14 29, Apparatus of claim 28 wherein at least some of said images are video images. 
15 

16 30. Apparatus of claim 28 wherein said images are arranged as a polyhedron, the 

17 orientation of said polyhedron being controlled by said control device. 
18 

19 31. Apparatus of claim 28 wherein said images are arranged as a band of images. 
20 

21 32. Apparatus of claim 28 wherein said display screen comprises primary and 

22 secondary regions, said plurality of images being displayed in said secondary region, at 
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1 least some of said images within said plurality of images corresponding to control options 

2 of what is to be displayed in said primary region. 
3 

4 33. Apparatus comprising: 

5 an image display device; and 

6 a remote control device for being held in a user's hand and for controlling the 

7 image displayed on said image display device, said remote control device detecting the 

8 angle or position at which the user is holding said remote control device. 
9 

10 34. Apparatus of claim 33 wherein said remote control device further comprises two 

1 1 motion sensors and a circuit for calculating the difference between the motion of the two 

12 motion sensors. 
13 

14 35. Apparatus of claim 33 wherein said remote control device further comprises a 

15 gyroscope for sensing motion and/or position of said remote control device. 
16 

17 36. Apparatus of claim 33 wherein said remote control device wherein the position 

18 and/or motion of said remote control device is sensed using a global position system. 
19 

20 37. Apparatus of claim 33 wherein said remote control device determines said angle 

2 1 or position without reference to the position of a fixed non-moving structure 

22 mechanically coupled to a moving structure. 
23 
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1 38. Apparatus of claim 33 wherein said display device comprises a screen for 

2 displaying an image, said image comprising a plurality effaces, the orientation of said 

3 faces being changed in response to the position of said remote control device. 

4 

5 39. Apparams of claim 38 wherein said plurality of faces form at least one 

6 polyhedron. 

7 

8 40. Apparatus of claim 38 wherein said plurality of faces forms a band of images. 
9 

10 41. Apparatus of claim 38 further comprising a graphics pipeline for providing an 

1 1 image on each of the faces within said plurality. 
12 

13 42. A remote control device for being held in a user's hand, said remote control 

14 device comprising: 

15 first and second motion sensors; and 

16 a circuit for determining the motion and/or position of said remote control device 

17 based on the motions sensed by said first and second motion sensors, said circuit 

18 providing a signal indicative of the motion and/or position in which the remote control 

19 device is being held. 
20 

21 43. A remote control device for being held in a user's hand, said remote control 

22 device comprising: 

23 a gyroscope; and 
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1 a circuit for determining the motion and/or position of said remote control device 

2 based on motion sensed by said gyroscope, said circuit providing a signal indicative of 

3 the motion and/or position in which the remote control device is being held. 
4 

5 44. A remote control device for being held in a user's hand, said remote control 

6 device comprising: 

7 first and second position sensors; and 

8 a circuit for determining the motion and/or position of said remote control device 

9 based on the positions sensed by said first and second position sensors, said circuit 

10 providing a signal indicative of the motion and/or position in which the remote control 

1 1 device is being held, 
12 

13 45. A remote control device comprising a member for being held by the hand of a 

14 user, said remote control device providing a signal indicative of the motion and/or 

15 position in which said member is being held without said member being mechanically 

16 coupled to a second structure and generating a signal indicative of the relative motion 

1 7 between said member and said second structure. 
18 

19 46. A method for using a remote control device, said remote control device 

20 comprising: 

21 first and second motion sensors; and 

22 a circuit for determining the motion and/or position of said remote control device 

23 based on the motions sensed by said first and second motion sensors, said circuit 
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1 providing a signal indicative of the motion and/or position in w^hich the remote control 

2 device is being held, said method comprising: 

3 moving said remote control device; and 

4 causing said circuit to calculate the motion and/or position of said remote control 

5 device; and 

6 providing a signal mdicative of said motion and/or position. 
7 

8 47. A method for using a remote control device, said remote control device 

9 comprising: 

10 a gyroscope; and 

1 1 a circuit for determining the motion and/or position of said remote control device 

12 based on motion sensed by said gyroscope, said circuit providing a signal indicative of 

13 the motion and/or position in which the remote control device is being held, 

14 said method comprising the step of moving said remote control device; and 

1 5 causing said circuit to emit a signal indicative of the motion and/or position of 

1 6 said remote control device. 
17 

18 48. A method for using a remote control device, said remote control device 

19 comprising first and second position sensors, said method comprising the steps of: 

20 calculating the position and/or motion of said remote control device in response to 

2 1 the position sensed by said position sensors; and 

22 providing a signal indicative of said calculated position and/or motion. 
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2 49. A remote control device comprising a member for being held by the hand of a 

3 user, said remote control device providing a signal indicative of the motion and/or 

4 position in which said member is being held without said member being mechanically 

5 coupled to a second structure and generating a signal indicative of the relative motion 

6 between said member and said second structure. 
7 

8 50. A method for using a remote control device, said remote control device 

9 comprising: 

10 a member for being held by the hand of a user, said remote control device 

1 1 providing a signal mdicative of the motion and/or position in which said member is being 

12 held without said member being mechanically coupled to a second structure and 

13 generating a signal indicative of the relative motion between said member and said 

14 second structure, 

15 said method comprising: 

16 grasping said remote control device and moving said remote control device to 

17 thereby cause said remote control device to generate a signal indicating the motion and/or 

18 position of said remote control device. 
19 
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